{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPUs\n",
    "\n",
    "In the introduction to this book we discussed the rapid growth of computation over the past two decades. In a nutshell, GPU performance has increased by a factor of 1000 every decade since 2000. This offers great opportunity but it also suggests a significant need to provide such performance.\n",
    "\n",
    "|Decade|Dataset|Memory|Floating Point Calculations per Second|\n",
    "|:--|:-|:-|:-|\n",
    "|1970|100 (Iris)|1 KB|100 KF (Intel 8080)|\n",
    "|1980|1 K (House prices in Boston)|100 KB|1 MF (Intel 80186)|\n",
    "|1990|10 K (optical character recognition)|10 MB|10 MF (Intel 80486)|\n",
    "|2000|10 M (web pages)|100 MB|1 GF (Intel Core)|\n",
    "|2010|10 G (advertising)|1 GB|1 TF (NVIDIA C2050)|\n",
    "|2020|1 T (social network)|100 GB|1 PF (NVIDIA DGX-2)|\n",
    "\n",
    "In this section we begin to discuss how to harness this compute performance for your research. First by using single GPUs and at a later point, how to use multiple GPUs and multiple servers (with multiple GPUs). You might have noticed that PyTorch Tensor looks almost identical to NumPy. But there are a few crucial differences. One of the key features that differentiates PyTorch from NumPy is its support for diverse hardware devices.\n",
    "\n",
    "In PyTorch, every tensor has a device. As we will discover, this just indicates that the computation is being executed on this device(earlier we saw this was always a CPU). Other deivces might be various GPUs. Things can get even hairier when we deploy jobs across multiple servers. By assigning arrays to devices intelligently, we can minimize the time spent transferring data between devices. For example, when training neural networks on a server with a GPU, we typically prefer for the model’s parameters to live on the GPU.\n",
    "\n",
    "In short, for complex neural networks and large-scale data, using only CPUs for computation may be inefficient. In this section, we will discuss how to use a single NVIDIA GPU for calculations. First, make sure you have at least one NVIDIA GPU installed. Then, [download CUDA](https://developer.nvidia.com/cuda-downloads) and follow the prompts to set the appropriate path. Once these preparations are complete, the `nvidia-smi` command can be used to view the graphics card information.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jun  3 15:18:03 2019       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |\n",
      "| 53%   63C    P2    78W / 250W |  10755MiB / 11178MiB |     33%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   1  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |\n",
      "|  0%   31C    P0    63W / 250W |      0MiB / 11178MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   2  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |\n",
      "| 27%   55C    P2   150W / 250W |   9953MiB / 11178MiB |     92%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   3  GeForce GTX 108...  Off  | 00000000:07:00.0 Off |                  N/A |\n",
      "| 67%   68C    P2   217W / 250W |  10977MiB / 11178MiB |     98%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   4  GeForce GTX 108...  Off  | 00000000:0B:00.0 Off |                  N/A |\n",
      "|  0%   38C    P0    61W / 250W |      0MiB / 11178MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   5  GeForce GTX 108...  Off  | 00000000:0C:00.0 Off |                  N/A |\n",
      "| 90%   79C    P2   155W / 250W |   6947MiB / 11178MiB |     97%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   6  GeForce GTX 108...  Off  | 00000000:0D:00.0 Off |                  N/A |\n",
      "| 28%   55C    P2   142W / 250W |   5855MiB / 11178MiB |    100%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "|   7  GeForce GTX 108...  Off  | 00000000:0E:00.0 Off |                  N/A |\n",
      "| 52%   63C    P2   287W / 250W |   6399MiB / 11178MiB |     98%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                       GPU Memory |\n",
      "|  GPU       PID   Type   Process name                             Usage      |\n",
      "|=============================================================================|\n",
      "|    0     39535      C   python                                     10745MiB |\n",
      "|    2      6989      C   python3                                     9943MiB |\n",
      "|    3     55224      C   python                                     10967MiB |\n",
      "|    5     31002      C   python                                      6937MiB |\n",
      "|    6      7897      C   python                                      5845MiB |\n",
      "|    7     39785      C   python                                      6389MiB |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to confirm that the GPU version of PyTorch is installed. If a CPU version of PyTorch is already installed, we need to uninstall it first. For example, use the `pip uninstall torch` command, then install the corresponding PyTorch version according to the CUDA version. Assuming you have CUDA 10.0 installed, you can install the PyTorch version that supports CUDA 10.0. To run the programs in this section, you need at least two GPUs.\n",
    "\n",
    "Note that this might be extravagant for most desktop computers but it is easily available in the cloud, e.g. by using the AWS EC2 multi-GPU instances. Almost all other sections do *not* require multiple GPUs. Instead, this is simply to illustrate how data flows between different devices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Devices\n",
    "\n",
    "PyTorch can specify devices, such as CPUs and GPUs, for storage and calculation. By default, PyTorch creates data in the main memory and then uses the CPU to calculate it. In PyTorch, the CPU and GPU can be indicated by `type = 'cpu'` and `type = 'gpu'`. It should be noted that `torch.device('cpu')` (or any integer in the parentheses) means all physical CPUs and memory. This means that PyTorch's calculations will try to use all CPU cores. However, `torch.device('cuda:0')` only represents one graphic card and the corresponding graphic memory. If there are multiple GPUs, we use `torch.device('cuda:i')` to represent the $i$-th GPU ($i$ starts from 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(device(type='cpu'),\n",
       " <torch.cuda.device at 0x7f25e54c0400>,\n",
       " <torch.cuda.device at 0x7f25e5404780>)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "torch.device('cpu'), torch.cuda.device('cuda'), torch.cuda.device('cuda:1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors and GPUs\n",
    "\n",
    "By default, Tensor objects are created on the CPU. Therefore, we will see the `type='cpu'` identifier each time we use `.device` attribute.  We can use the `device` property of Tensor to view the device where the Tensor is located. It is important to note that whenever we want to operate on multiple terms they need to be in the same device. For instance, if we sum two variables, we need to make sure that both arguments are on the same device - otherwise PyTorch would not know where to store the result or even how to decide where to perform the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor([1, 2, 3])\n",
    "x.device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storage on the GPU\n",
    "\n",
    "There are several ways to store a Tensor on the GPU. For example, we can specify a storage device with the `device` parameter when creating a Tensor. Next, we create the Tensor variable `a` on `gpu(0)`. Notice that when printing `a`, the device information becomes `device='cuda:0'`. The Tensor created on a GPU only consumes the memory of this GPU. We can use the `nvidia-smi` command to view GPU memory usage. In general, we need to make sure we do not create data that exceeds the GPU memory limit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='cuda:0')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.ones((2, 3), device=torch.device('cuda:0'))\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming you have at least two GPUs, the following code will create a random array on `gpu(1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.9060e-03, -2.4239e-01, -4.1872e-01],\n",
       "        [ 2.8501e+00,  4.0416e-01, -2.9519e-01]], device='cuda:1')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.randn(size=(2, 3), device=torch.device('cuda:1'))\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copying\n",
    "\n",
    "If we want to compute $\\mathbf{x} + \\mathbf{y}$ we need to decide where to perform this operation. For instance, we can transfer $\\mathbf{x}$ to `gpu(1)` and perform the operation there. **Do not** simply add `x + y` since this will result in an exception. The runtime engine wouldn't know what to do, it cannot find data on the same device and it fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"98pt\" version=\"1.1\" viewBox=\"0 0 236 98\" width=\"236pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.453125 0.390625 L 1.21875 0.5 C 1.25 0.738281 1.335938 0.910156 1.484375 1.015625 C 1.679688 1.160156 1.953125 1.234375 2.296875 1.234375 C 2.660156 1.234375 2.941406 1.160156 3.140625 1.015625 C 3.335938 0.867188 3.472656 0.664062 3.546875 0.40625 C 3.585938 0.238281 3.601562 -0.0976562 3.59375 -0.609375 C 3.25 -0.203125 2.820312 0 2.3125 0 C 1.664062 0 1.164062 -0.226562 0.8125 -0.6875 C 0.46875 -1.15625 0.296875 -1.710938 0.296875 -2.359375 C 0.296875 -2.804688 0.375 -3.21875 0.53125 -3.59375 C 0.695312 -3.96875 0.929688 -4.253906 1.234375 -4.453125 C 1.535156 -4.660156 1.894531 -4.765625 2.3125 -4.765625 C 2.863281 -4.765625 3.316406 -4.546875 3.671875 -4.109375 L 3.671875 -4.671875 L 4.40625 -4.671875 L 4.40625 -0.640625 C 4.40625 0.0859375 4.328125 0.601562 4.171875 0.90625 C 4.023438 1.207031 3.789062 1.445312 3.46875 1.625 C 3.15625 1.800781 2.765625 1.890625 2.296875 1.890625 C 1.734375 1.890625 1.28125 1.765625 0.9375 1.515625 C 0.601562 1.265625 0.441406 0.890625 0.453125 0.390625 Z M 1.109375 -2.421875 C 1.109375 -1.804688 1.226562 -1.359375 1.46875 -1.078125 C 1.707031 -0.796875 2.007812 -0.65625 2.375 -0.65625 C 2.738281 -0.65625 3.046875 -0.796875 3.296875 -1.078125 C 3.546875 -1.359375 3.671875 -1.796875 3.671875 -2.390625 C 3.671875 -2.960938 3.539062 -3.394531 3.28125 -3.6875 C 3.03125 -3.976562 2.726562 -4.125 2.375 -4.125 C 2.019531 -4.125 1.71875 -3.976562 1.46875 -3.6875 C 1.226562 -3.40625 1.109375 -2.984375 1.109375 -2.421875 Z M 1.109375 -2.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 1.78125 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4.0625 C 1.476562 -4.300781 1.664062 -4.476562 1.875 -4.59375 C 2.09375 -4.707031 2.359375 -4.765625 2.671875 -4.765625 C 3.066406 -4.765625 3.414062 -4.660156 3.71875 -4.453125 C 4.019531 -4.253906 4.25 -3.96875 4.40625 -3.59375 C 4.5625 -3.21875 4.640625 -2.8125 4.640625 -2.375 C 4.640625 -1.894531 4.550781 -1.460938 4.375 -1.078125 C 4.207031 -0.691406 3.960938 -0.394531 3.640625 -0.1875 C 3.316406 0.0078125 2.972656 0.109375 2.609375 0.109375 C 2.347656 0.109375 2.113281 0.0507812 1.90625 -0.0625 C 1.695312 -0.175781 1.523438 -0.316406 1.390625 -0.484375 L 1.390625 1.78125 Z M 1.3125 -2.3125 C 1.3125 -1.707031 1.429688 -1.257812 1.671875 -0.96875 C 1.921875 -0.6875 2.21875 -0.546875 2.5625 -0.546875 C 2.90625 -0.546875 3.203125 -0.691406 3.453125 -0.984375 C 3.710938 -1.285156 3.84375 -1.75 3.84375 -2.375 C 3.84375 -2.96875 3.71875 -3.410156 3.46875 -3.703125 C 3.226562 -4.003906 2.9375 -4.15625 2.59375 -4.15625 C 2.257812 -4.15625 1.960938 -3.992188 1.703125 -3.671875 C 1.441406 -3.359375 1.3125 -2.90625 1.3125 -2.3125 Z M 1.3125 -2.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 3.65625 0 L 3.65625 -0.6875 C 3.289062 -0.15625 2.796875 0.109375 2.171875 0.109375 C 1.898438 0.109375 1.644531 0.0546875 1.40625 -0.046875 C 1.164062 -0.160156 0.984375 -0.296875 0.859375 -0.453125 C 0.742188 -0.609375 0.664062 -0.800781 0.625 -1.03125 C 0.59375 -1.1875 0.578125 -1.4375 0.578125 -1.78125 L 0.578125 -4.671875 L 1.359375 -4.671875 L 1.359375 -2.078125 C 1.359375 -1.660156 1.378906 -1.382812 1.421875 -1.25 C 1.460938 -1.039062 1.5625 -0.875 1.71875 -0.75 C 1.882812 -0.632812 2.085938 -0.578125 2.328125 -0.578125 C 2.566406 -0.578125 2.789062 -0.632812 3 -0.75 C 3.207031 -0.875 3.351562 -1.039062 3.4375 -1.25 C 3.519531 -1.457031 3.5625 -1.765625 3.5625 -2.171875 L 3.5625 -4.671875 L 4.359375 -4.671875 L 4.359375 0 Z M 3.65625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.109375 1.890625 C 1.671875 1.335938 1.300781 0.695312 1 -0.03125 C 0.695312 -0.769531 0.546875 -1.535156 0.546875 -2.328125 C 0.546875 -3.023438 0.660156 -3.695312 0.890625 -4.34375 C 1.148438 -5.082031 1.554688 -5.816406 2.109375 -6.546875 L 2.671875 -6.546875 C 2.316406 -5.941406 2.082031 -5.507812 1.96875 -5.25 C 1.789062 -4.84375 1.648438 -4.421875 1.546875 -3.984375 C 1.421875 -3.429688 1.359375 -2.878906 1.359375 -2.328125 C 1.359375 -0.921875 1.796875 0.484375 2.671875 1.890625 Z M 2.109375 1.890625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.375 -3.171875 C 0.375 -3.929688 0.453125 -4.546875 0.609375 -5.015625 C 0.765625 -5.484375 0.992188 -5.84375 1.296875 -6.09375 C 1.609375 -6.34375 2 -6.46875 2.46875 -6.46875 C 2.820312 -6.46875 3.128906 -6.394531 3.390625 -6.25 C 3.648438 -6.113281 3.863281 -5.914062 4.03125 -5.65625 C 4.195312 -5.394531 4.328125 -5.078125 4.421875 -4.703125 C 4.523438 -4.328125 4.578125 -3.816406 4.578125 -3.171875 C 4.578125 -2.421875 4.5 -1.8125 4.34375 -1.34375 C 4.1875 -0.882812 3.953125 -0.523438 3.640625 -0.265625 C 3.335938 -0.015625 2.945312 0.109375 2.46875 0.109375 C 1.851562 0.109375 1.367188 -0.113281 1.015625 -0.5625 C 0.585938 -1.09375 0.375 -1.960938 0.375 -3.171875 Z M 1.1875 -3.171875 C 1.1875 -2.117188 1.304688 -1.414062 1.546875 -1.0625 C 1.796875 -0.71875 2.101562 -0.546875 2.46875 -0.546875 C 2.832031 -0.546875 3.140625 -0.71875 3.390625 -1.0625 C 3.640625 -1.414062 3.765625 -2.117188 3.765625 -3.171875 C 3.765625 -4.234375 3.640625 -4.9375 3.390625 -5.28125 C 3.140625 -5.632812 2.832031 -5.8125 2.46875 -5.8125 C 2.101562 -5.8125 1.8125 -5.660156 1.59375 -5.359375 C 1.320312 -4.960938 1.1875 -4.234375 1.1875 -3.171875 Z M 1.1875 -3.171875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"M 1.109375 1.890625 L 0.546875 1.890625 C 1.421875 0.484375 1.859375 -0.921875 1.859375 -2.328125 C 1.859375 -2.878906 1.796875 -3.425781 1.671875 -3.96875 C 1.566406 -4.40625 1.425781 -4.828125 1.25 -5.234375 C 1.132812 -5.492188 0.898438 -5.929688 0.546875 -6.546875 L 1.109375 -6.546875 C 1.660156 -5.816406 2.066406 -5.082031 2.328125 -4.34375 C 2.554688 -3.695312 2.671875 -3.023438 2.671875 -2.328125 C 2.671875 -1.535156 2.519531 -0.769531 2.21875 -0.03125 C 1.914062 0.695312 1.546875 1.335938 1.109375 1.890625 Z M 1.109375 1.890625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-7\" overflow=\"visible\">\n",
       "<path d=\"M 3.359375 0 L 2.5625 0 L 2.5625 -5.046875 C 2.375 -4.859375 2.125 -4.671875 1.8125 -4.484375 C 1.5 -4.304688 1.222656 -4.175781 0.984375 -4.09375 L 0.984375 -4.859375 C 1.421875 -5.054688 1.804688 -5.300781 2.140625 -5.59375 C 2.472656 -5.894531 2.707031 -6.1875 2.84375 -6.46875 L 3.359375 -6.46875 Z M 3.359375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-8\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -1.703125 L 4.421875 -1.609375 C 4.335938 -1.066406 4.117188 -0.644531 3.765625 -0.34375 C 3.410156 -0.0390625 2.976562 0.109375 2.46875 0.109375 C 1.832031 0.109375 1.320312 -0.0976562 0.9375 -0.515625 C 0.550781 -0.929688 0.359375 -1.53125 0.359375 -2.3125 C 0.359375 -2.820312 0.441406 -3.265625 0.609375 -3.640625 C 0.773438 -4.015625 1.023438 -4.296875 1.359375 -4.484375 C 1.703125 -4.671875 2.078125 -4.765625 2.484375 -4.765625 C 2.984375 -4.765625 3.394531 -4.632812 3.71875 -4.375 C 4.039062 -4.125 4.25 -3.765625 4.34375 -3.296875 L 3.578125 -3.171875 C 3.503906 -3.484375 3.375 -3.71875 3.1875 -3.875 C 3 -4.039062 2.773438 -4.125 2.515625 -4.125 C 2.109375 -4.125 1.78125 -3.976562 1.53125 -3.6875 C 1.289062 -3.40625 1.171875 -2.957031 1.171875 -2.34375 C 1.171875 -1.707031 1.289062 -1.25 1.53125 -0.96875 C 1.769531 -0.6875 2.082031 -0.546875 2.46875 -0.546875 C 2.78125 -0.546875 3.039062 -0.640625 3.25 -0.828125 C 3.457031 -1.015625 3.585938 -1.304688 3.640625 -1.703125 Z M 3.640625 -1.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.296875 -2.328125 C 0.296875 -3.191406 0.535156 -3.832031 1.015625 -4.25 C 1.421875 -4.59375 1.910156 -4.765625 2.484375 -4.765625 C 3.128906 -4.765625 3.65625 -4.554688 4.0625 -4.140625 C 4.46875 -3.722656 4.671875 -3.144531 4.671875 -2.40625 C 4.671875 -1.800781 4.578125 -1.328125 4.390625 -0.984375 C 4.210938 -0.640625 3.953125 -0.367188 3.609375 -0.171875 C 3.265625 0.015625 2.890625 0.109375 2.484375 0.109375 C 1.828125 0.109375 1.296875 -0.0976562 0.890625 -0.515625 C 0.492188 -0.941406 0.296875 -1.546875 0.296875 -2.328125 Z M 1.109375 -2.328125 C 1.109375 -1.734375 1.238281 -1.285156 1.5 -0.984375 C 1.757812 -0.691406 2.085938 -0.546875 2.484375 -0.546875 C 2.878906 -0.546875 3.207031 -0.691406 3.46875 -0.984375 C 3.726562 -1.285156 3.859375 -1.742188 3.859375 -2.359375 C 3.859375 -2.929688 3.726562 -3.367188 3.46875 -3.671875 C 3.207031 -3.972656 2.878906 -4.125 2.484375 -4.125 C 2.085938 -4.125 1.757812 -3.972656 1.5 -3.671875 C 1.238281 -3.378906 1.109375 -2.929688 1.109375 -2.328125 Z M 1.109375 -2.328125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-10\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 1.796875 L 0.46875 1.0625 C 0.644531 1.101562 0.796875 1.125 0.921875 1.125 C 1.097656 1.125 1.238281 1.09375 1.34375 1.03125 C 1.445312 0.976562 1.535156 0.898438 1.609375 0.796875 C 1.648438 0.710938 1.726562 0.515625 1.84375 0.203125 C 1.863281 0.160156 1.890625 0.0976562 1.921875 0.015625 L 0.140625 -4.671875 L 1 -4.671875 L 1.96875 -1.96875 C 2.09375 -1.625 2.207031 -1.265625 2.3125 -0.890625 C 2.394531 -1.242188 2.5 -1.597656 2.625 -1.953125 L 3.625 -4.671875 L 4.421875 -4.671875 L 2.640625 0.078125 C 2.453125 0.585938 2.304688 0.941406 2.203125 1.140625 C 2.054688 1.398438 1.894531 1.585938 1.71875 1.703125 C 1.539062 1.828125 1.320312 1.890625 1.0625 1.890625 C 0.914062 1.890625 0.75 1.859375 0.5625 1.796875 Z M 0.5625 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-11\" overflow=\"visible\">\n",
       "<path d=\"M 2.328125 -0.703125 L 2.4375 -0.015625 C 2.207031 0.0351562 2.007812 0.0625 1.84375 0.0625 C 1.550781 0.0625 1.328125 0.015625 1.171875 -0.078125 C 1.015625 -0.171875 0.898438 -0.289062 0.828125 -0.4375 C 0.765625 -0.582031 0.734375 -0.890625 0.734375 -1.359375 L 0.734375 -4.046875 L 0.15625 -4.046875 L 0.15625 -4.671875 L 0.734375 -4.671875 L 0.734375 -5.828125 L 1.53125 -6.296875 L 1.53125 -4.671875 L 2.328125 -4.671875 L 2.328125 -4.046875 L 1.53125 -4.046875 L 1.53125 -1.328125 C 1.53125 -1.097656 1.539062 -0.953125 1.5625 -0.890625 C 1.59375 -0.828125 1.640625 -0.773438 1.703125 -0.734375 C 1.765625 -0.691406 1.851562 -0.671875 1.96875 -0.671875 C 2.0625 -0.671875 2.179688 -0.679688 2.328125 -0.703125 Z M 2.328125 -0.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M -0.015625 0 L 2.015625 -2.375 L 0.859375 -4.671875 L 1.734375 -4.671875 L 2.125 -3.84375 C 2.269531 -3.53125 2.398438 -3.226562 2.515625 -2.9375 L 3.875 -4.671875 L 4.84375 -4.671875 L 2.875 -2.3125 L 4.0625 0 L 3.171875 0 L 2.71875 -0.953125 C 2.613281 -1.148438 2.5 -1.398438 2.375 -1.703125 L 0.984375 0 Z M -0.015625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.171875 0 L 0.296875 -0.609375 L 2.8125 -3.4375 C 2.988281 -3.632812 3.1875 -3.84375 3.40625 -4.0625 C 3.03125 -4.019531 2.757812 -4 2.59375 -4 L 0.859375 -4 L 1 -4.671875 L 4.609375 -4.671875 L 4.5 -4.171875 L 1.96875 -1.328125 C 1.832031 -1.160156 1.617188 -0.9375 1.328125 -0.65625 C 1.773438 -0.6875 2.078125 -0.703125 2.234375 -0.703125 L 4.078125 -0.703125 L 3.9375 0 Z M 0.171875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 0 1.796875 L 0.046875 1.0625 C 0.210938 1.101562 0.375 1.125 0.53125 1.125 C 0.6875 1.125 0.816406 1.085938 0.921875 1.015625 C 1.046875 0.921875 1.1875 0.738281 1.34375 0.46875 L 1.59375 0.015625 L 0.8125 -4.671875 L 1.59375 -4.671875 L 1.953125 -2.3125 C 2.023438 -1.84375 2.082031 -1.375 2.125 -0.90625 L 4.21875 -4.671875 L 5.046875 -4.671875 L 2.078125 0.625 C 1.785156 1.132812 1.523438 1.472656 1.296875 1.640625 C 1.078125 1.804688 0.828125 1.890625 0.546875 1.890625 C 0.359375 1.890625 0.175781 1.859375 0 1.796875 Z M 0 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.5625 -1.046875 L 2.5625 -2.8125 L 0.8125 -2.8125 L 0.8125 -3.546875 L 2.5625 -3.546875 L 2.5625 -5.296875 L 3.3125 -5.296875 L 3.3125 -3.546875 L 5.0625 -3.546875 L 5.0625 -2.8125 L 3.3125 -2.8125 L 3.3125 -1.046875 Z M 2.5625 -1.046875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 142.601562 54.199219 L 237.800781 54.199219 L 237.800781 150 L 142.601562 150 Z M 142.601562 54.199219 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"35.19219\" xlink:href=\"#glyph0-1\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"40.19799\" xlink:href=\"#glyph0-2\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"45.20379\" xlink:href=\"#glyph0-3\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"50.20959\" xlink:href=\"#glyph0-4\" y=\"89.352783\"/>\n",
       "  <use x=\"53.20659\" xlink:href=\"#glyph0-5\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"58.21239\" xlink:href=\"#glyph0-6\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<path d=\"M 178.414062 105.132812 C 183.394531 110.113281 183.394531 118.1875 178.414062 123.164062 C 173.4375 128.144531 165.363281 128.144531 160.382812 123.164062 C 155.40625 118.1875 155.40625 110.113281 160.382812 105.132812 C 165.363281 100.15625 173.4375 100.15625 178.414062 105.132812 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"25.15\" xlink:href=\"#glyph1-1\" y=\"64.002783\"/>\n",
       "</g>\n",
       "<path d=\"M 281.601562 54.199219 L 376.800781 54.199219 L 376.800781 150 L 281.601562 150 Z M 281.601562 54.199219 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"174.19219\" xlink:href=\"#glyph0-1\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"179.19799\" xlink:href=\"#glyph0-2\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"184.20379\" xlink:href=\"#glyph0-3\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"189.20959\" xlink:href=\"#glyph0-4\" y=\"89.352783\"/>\n",
       "  <use x=\"192.20659\" xlink:href=\"#glyph0-7\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"197.21239\" xlink:href=\"#glyph0-6\" y=\"89.352783\"/>\n",
       "</g>\n",
       "<path d=\"M 317.414062 105.132812 C 322.394531 110.113281 322.394531 118.1875 317.414062 123.164062 C 312.4375 128.144531 304.363281 128.144531 299.382812 123.164062 C 294.40625 118.1875 294.40625 110.113281 299.382812 105.132812 C 304.363281 100.15625 312.4375 100.15625 317.414062 105.132812 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"164.15\" xlink:href=\"#glyph1-2\" y=\"64.002783\"/>\n",
       "</g>\n",
       "<path d=\"M 359.015625 105.132812 C 363.996094 110.113281 363.996094 118.1875 359.015625 123.164062 C 354.035156 128.144531 345.964844 128.144531 340.984375 123.164062 C 336.003906 118.1875 336.003906 110.113281 340.984375 105.132812 C 345.964844 100.15625 354.035156 100.15625 359.015625 105.132812 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"205.75\" xlink:href=\"#glyph1-3\" y=\"64.002783\"/>\n",
       "</g>\n",
       "<path d=\"M 338.214844 64.132812 C 343.195312 69.113281 343.195312 77.1875 338.214844 82.164062 C 333.238281 87.144531 325.164062 87.144531 320.183594 82.164062 C 315.207031 77.1875 315.207031 69.113281 320.183594 64.132812 C 325.164062 59.15625 333.238281 59.15625 338.214844 64.132812 \" style=\"fill-rule:nonzero;fill:rgb(41.567993%,75.294495%,98.823547%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"184.57207\" xlink:href=\"#glyph1-4\" y=\"23.002783\"/>\n",
       "</g>\n",
       "<path d=\"M 314.171875 102.777344 L 320.761719 89.785156 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<path d=\"M 322.570312 86.21875 L 319.421875 89.105469 L 322.097656 90.464844 Z M 322.570312 86.21875 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<path d=\"M 344.230469 102.777344 L 337.640625 89.785156 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<path d=\"M 335.828125 86.21875 L 336.300781 90.464844 L 338.976562 89.105469 Z M 335.828125 86.21875 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<path d=\"M 182.148438 114.148438 L 289.75 114.148438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<path d=\"M 293.75 114.148438 L 289.75 112.648438 L 289.75 115.648438 Z M 293.75 114.148438 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-142,-53)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"101.84463\" xlink:href=\"#glyph0-8\" y=\"56.952783\"/>\n",
       "  <use x=\"106.34463\" xlink:href=\"#glyph0-9\" y=\"56.952783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"111.35043\" xlink:href=\"#glyph0-2\" y=\"56.952783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"116.35623\" xlink:href=\"#glyph0-10\" y=\"56.952783\"/>\n",
       "  <use x=\"120.85623\" xlink:href=\"#glyph0-11\" y=\"56.952783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"123.35643\" xlink:href=\"#glyph0-9\" y=\"56.952783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"128.36223\" xlink:href=\"#glyph0-4\" y=\"56.952783\"/>\n",
       "  <use x=\"131.35923\" xlink:href=\"#glyph0-6\" y=\"56.952783\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename=\"../img/copyto.svg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of `copyto` as shown in image, in PyTorch we simply use `copy_` to copy the tensor to the target device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], device='cuda:0')\n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]], device='cuda:1')\n"
     ]
    }
   ],
   "source": [
    "z = x.cuda(1)\n",
    "print(x)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This copies the data to another device such that we can add them. Since $\\mathbf{y}$ lives on the second GPU we need to move $\\mathbf{x}$ there before we can add the two.\n",
    "\n",
    "Now that the data is on the same GPU (both $\\mathbf{z}$ and $\\mathbf{y}$ are), we can add them up. In such cases PyTorch places the result on the same device as its constituents. In our case that is `gpu(1)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0019, 0.7576, 0.5813],\n",
       "        [3.8501, 1.4042, 0.7048]], device='cuda:1')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y + z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine that your variable z already lives on your second GPU (gpu(1)). What happens if we call `z.copy_(z)`? It will make a copy and allocate new memory, even though that variable already lives on the desired device!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='cuda:1')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.copy_(z)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], device='cuda:0')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to note that, if the `device` of the source variable and the target variable are consistent, then the `.cuda` function causes the target variable and the source variable to share the memory of the source variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y is y.cuda(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `copy_` function mutates in place and no copy is made unlike MXNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y is y.copy_(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch Out\n",
    "\n",
    "People use GPUs to do machine learning because they expect them to be fast. But transferring variables between devices is slow. So we want you to be 100% certain that you want to do something slow before we let you do it. If PyTorch just did the copy automatically without crashing then you might not realize that you had written some slow code.\n",
    "\n",
    "Also, transferring data between devices (CPU, GPUs, other machines) is something that is *much slower* than computation. It also makes parallelization a lot more difficult, since we have to wait for data to be sent (or rather to be received) before we can proceed with more operations. This is why copy operations should be taken with great care. As a rule of thumb, many small operations are much worse than one big operation. Moreover, several operations at a time are much better than many single operations interspersed in the code (unless you know what you're doing). This is the case since such operations can block if one device has to wait for the other before it can do something else. It's a bit like ordering your coffee in a queue rather than pre-ordering it by phone and finding out that it's ready when you are.\n",
    "\n",
    "Lastly, when we print Tensor data or convert Tensor to NumPy format, if the data is not in main memory, PyTorch will copy it to the main memory first, resulting in additional transmission overhead. Even worse, it is now subject to the dreaded Global Interpreter Lock which makes everything wait for Python to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch and GPUs\n",
    "\n",
    "Similar to Tensors, PyTorch's model can specify devices through the `.to(device)` parameter during initialization. The following code initializes the model parameters on the GPU (we will see many more examples of how to run models on GPUs in the following, simply since they will become somewhat more compute intensive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = nn.Sequential(nn.Linear(3,1))\n",
    "net = net.to(device='cuda:0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the input is a Tensor on the GPU, PyTorch will calculate the result on the same GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.3683],\n",
       "        [1.3683]], device='cuda:0', grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us confirm that the model parameters are stored on the same GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[0.3571, 0.4750, 0.5428]], device='cuda:0', requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([-0.0066], device='cuda:0', requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "for param in net[0].parameters():\n",
    "    print(param)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In short, as long as all data and parameters are on the same device, we can learn models efficiently. In the following we will see several such examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* PyTorch can specify devices for storage and calculation, such as CPU or GPU. By default, PyTorch creates data in the main memory and then uses the CPU to calculate it.\n",
    "* PyTorch requires all input data for calculation to be **on the same device**, be it CPU or the same GPU.\n",
    "* You can lose significant performance by moving data without care. A typical mistake is as follows: computing the loss for every minibatch on the GPU and reporting it back to the user on the commandline (or logging it in a NumPy array) will trigger a global interpreter lock which stalls all GPUs. It is much better to allocate memory for logging inside the GPU and only move larger logs.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Try a larger computation task, such as the multiplication of large matrices, and see the difference in speed between the CPU and GPU. What about a task with a small amount of calculations?\n",
    "1. How should we read and write model parameters on the GPU?\n",
    "1. Measure the time it takes to compute 1000 matrix-matrix multiplications of $100 \\times 100$ matrices and log the matrix norm $\\mathrm{tr} M M^\\top$ one result at a time vs. keeping a log on the GPU and transferring only the final result.\n",
    "1. Measure how much time it takes to perform two matrix-matrix multiplications on two GPUs at the same time vs. in sequence on one GPU (hint - you should see almost linear scaling).\n",
    "\n",
    "## References\n",
    "\n",
    "[1] CUDA download address. https://developer.nvidia.com/cuda-downloads"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
